Lab window collaboration

ABSTRACT

This invention is a method for manipulating virtual objects displayed on a video conference broadcast by generating a computerized three dimensional image of an object to be superimposed on a first video broadcast signal from a local video camera for display on a remote video monitor, and superimposing the same object on a second video broadcast signal from a remote video camera for display on a local video monitor, grabbing a portion of the three dimensional image by placing a hand in close proximity to the portion of the image moving the hand while maintaining the hand in close proximity to the image and regenerating the three dimensional image to a new perspective view corresponding to the movement of the image with the hand to create the appearance that the hand is manipulating a virtual object displayed over the video broadcast signal.

BACKGROUND

[0001] This invention relates to video conferencing systems. Moreparticularly, this invention relates to computer-generated images forshared viewing and manipulation on a video conferencing broadcastdisplay monitor.

[0002] To enter a virtual reality environment, users must put on videodisplay goggles and body position sensors. Their hands and/or bodiesappear as a virtual image in the virtual reality environment, and canmanipulate virtual objects in that environment as seen through theirgoggles. Multiple users can appear before one another as virtual personsin a single virtual reality environment. Users from remote distantlocations can thereby have virtual meetings in that virtual environment,and view and manipulate virtual information and three-dimensionalobjects. Still, the participants cannot interact with each other as in areal face-to face meeting.

[0003] While virtual reality meetings may be common among networkedvirtual reality video games, video conferencing is the commonly acceptednorm for conducting face-to-face meetings of business people betweendistant remote locations. The participants see real images of otherparticipants at remote locations, but cannot readily share data ormanipulate virtual objects as in virtual reality environments.

[0004] Still, many multi-national corporations use video conferencingsystems to provide low-cost face-to-face meetings between colleagues atdistant locations. To enhance communications at those meetings, somevideo conferencing systems permit computer generated images orpresentations to be simultaneously broadcast to participants either in apop-up window or as an alternate switchable display on the videomonitors. Lately, enhancements to this have been provided for videoconferencing over the Internet that permits the manipulation by distantparticipants of computer-generated documents, spreadsheets or drawingsdisplayed in the separate pop-up window. While the sharing of suchinformation enhances the communicative exchange at such videoconferences, it does not replace actual meetings where detailedinformation concerning complex three-dimensional objects must be shared.

BRIEF SUMMARY OF THE INVENTION

[0005] The present invention meets the aforementioned need by mergingvideo conferencing and three-dimensional computer developmentapplications into a single collaboration tool. In one embodiment, theinvention is a system that includes at each location a large videomonitor with a touch screen, cameras associated with each monitor audioequipment, computer processing equipment and high bandwidthcommunication access. The components of the system cooperate to providea video conference broadcast with a three-dimensional computer-generatedimage superimposed on the video broadcast. This image appears as avirtual object in the plane of the monitor that can be manipulated inresponse to a participant at any location touching the screen near theobject to “grab” and move the object.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006]FIG. 1 is a schematic of one embodiment of a system, in accordancewith the invention.

[0007]FIG. 2 is an illustration of a video display monitor with avirtual object.

DETAILED DESCRIPTION

[0008]FIG. 1 depicts a preferred embodiment of the system of the presentinvention. A local video conferencing system 10 is connected via a highbandwidth communication network 12 to a remote location videoconferencing system 14 in a peer-to-peer model. The video conferencingsystems at either location include video display monitors 16 and 18,video cameras 20 and 22, touch screen input devices 24 and 26, which areconnected to computer processing systems 28 and 30. The computerprocessing systems include video input cards 32 and 34, and they alsoinclude other input devices such as computer mouse 36 and 38. Otherinput devices may include joysticks, keyboards, track balls or videogesture recognition systems. The system may include voice recognitionsystems for integrated operation with voice commands.

[0009] The cameras and display, in one sense, act as a standard videoconferencing equipment to broadcast the video image for display at themonitor of the remote location. However, with the cameras positionedover large monitors at each location with the appropriate camera lensfocal length to project a life-size image, the video monitor appears asa window into the conference room at the remote location.

[0010] The system depicted in FIG. 1 is an overview of the videoconferencing systems connected in a peer-to-peer model. Alternatively,the systems may be connected to a high bandwidth network in aclient/server model, or other models of distributed processing. Thesoftware for controlling the virtual object may reside in the server onthe network, locally in each peer, or in other distributions dependingon the system processing model used to implement the invention. Suchvarious distributed processing models have implemented for interactivenetworked video games.

[0011] The computer processor systems for controlling the imageprocessing, wherever residing, have suitable software for generatingthree-dimensional images superimposed or overlaying the video broadcastimage from the video cameras. As used herein, when a computer-generatedimage overlays or is superimposed on the video image, or vice versa, itshould be understood that this refers to the apparent relationshipbetween the computer-generated image and the video broadcast. Thecomputer-generated image appears to float as a solid object in front ofthe video conference broadcast image. For example, this can beaccomplished by generating a computer-generated image over a dark purplebackground. The video broadcast signal from the camera can then bedigitized on a video input card and overlay the dark purple backgroundof the three-dimensional computer-generated image with the result thatthe computer image appears to float over the video broadcast signal.This may also be accomplished by other techniques as are known to thoseof ordinary skill in the art.

[0012] Any suitable software language may be useful for this invention.For example, “Open GL” and “Direct X” are two such graphics programminglanguages that may be useful. Also, other higher-level languages thatsupport a variety of applications, such as “SGI's Open Inventor,” may beused.

[0013]FIG. 2 depicts the image that may be viewed on a video monitor 16.The video monitor displays a computer generated three-dimensional image40 overlaying the video broadcast of a scientist 42 at a videoconferencing system in a remote location. This three-dimensional objectmay be, for example, a three-dimensional rendering of a drug moleculeunder study by a group of scientists. The drug molecule 40 appears as avirtual three-dimensional object floating in space in the plane of thevideo display in front of the scientist. The video monitor appears toprovide a “window” into the laboratory of the scientist and provides anaugmented reality view by superimposing virtual objects on that window.

[0014] To enhance the reality of the displayed video broadcast to moreresemble a “window” into the other room, it is preferred to make thevideo monitor as large as possible. Ideally, the monitor may be a42-inch, or larger, flat panel video display hung on a wall at about theheight of a typical window. Also, the camera is preferable positionedand has a suitable focal length so that the broadcast view of the roomand participants therein appear life-size to viewers at the remotelocation. This gives the illusion that the participants are indeedstanding behind the “window” in the next room.

[0015] Participants at any location may readily manipulate the virtualobject 40. The hand 44 of the scientist, as shown in FIG. 2, appears tobe grabbing the virtual object 40. When his hand is in contact with thetouch screen input device near the virtual object, the computer systemrecognizes his touch as grabbing that portion of the object in closeproximity to his touch. When the scientist moves his hand along thetouch screen, the computer system moves, or rotates, the virtual objectso that the touch portion of the object tracks along with the movementof the scientist's hand. The system may be set up to reposition theimage for every movement of the hand of at least 0.5 centimeters toavoid shakiness. The resolution of movement should be set for theappropriate input device used by the scientist, however.

[0016] Likewise, a second scientist at a second location remote from afirst location viewing the image as seen in FIG. 2, can reach out andgrab the virtual object and move it as well. Scientist 42 would see thesame virtual object as the scientist at the first local location, andwould see the same object moved by the movement of the secondscientist's hand on the touch screen.

[0017] In initiating the conference, the data for the virtual image canbe transmitted from a computer at the first location to the computer atthe second location. Preferably, in a peer-to-peer system model, themovement of the object is synchronized between the local computers. Eachcomputer sensing a local manipulation of the object may re-render theobject locally corresponding to the new position and/or rotation, andtransmit the new position and/or rotation information to the computer atthe distant location for that distant computer to re-render the virtualobject with the new position and/or rotation.

[0018] In systems utilizing a central server, the central server couldinitially transmit the model for the virtual image to each localcomputer. As changes are made to the position and/or rotation of theobject, each local computer transmits the positional information to thecentral server as well as to the other local computers for re-renderingthe virtual image.

[0019] The system may provide a variety of different perspective viewsto the different locations as desired by the participants. For example,the system may present identical perspective views to each location, sothat each participant sees the same side of the object with left-rightelements correctly situated. The system may present mirror image viewsto each location, with left-right elements transposed but with the sameside of the object seem by each participant. This allows theparticipants to touch the same portion of the object by apparentlytouching the same opposing portion of the “window.” Or the system maypresent opposite perspectives views of the object that recreates theactual front-side and rear-side views of a real three-dimensional objectfloating between the respective participants. Preferably, the systemprovides the user with the option to select the perspective view mostdesirable for the specific application or object they are viewing.

[0020] According to the present invention, this method for manipulatingvirtual objects displayed on a video conference broadcast at a local anda remote location includes generating with a remote processor a threedimensional image of an object superimposed on a first video broadcastsignal from a local video camera 40 for display on a remote videomonitor, and superimposing a corresponding image on a second videobroadcast signal from a remote video camera for display on a local videomonitor. This method includes grabbing a portion of the image by placinga real object in close proximity to the portion of the image to activatea touch sensitive screen, and then moving the real object whilemaintaining the real object in active coupling with the touch sensitivescreen. The three dimensional image is regenerated by the computer to anew perspective view that corresponds to the movement or new location ofthe real object. This creates the appearance that the real object ismanipulating the virtual object. Preferably, the real object is aperson's fingers and/or hand and the person is located in front of oneof either of the remote or local video monitor and within view of one ofthe remote or local video cameras. Nonetheless, the real object couldeasily be a stick, a pen, or other pointing device.

[0021] The method allows for the natural manipulation of the virtualobject as though it were a real three dimensional object floating in the“window” between the two conference rooms. The method allows control bya person at either location at any time. The system may receiveconflicting inputs as to how to move or manipulate the virtual object.In those situations, social conventions and etiquette will dictate howthe virtual object or computer-generated image is manipulated. In otherwords, one person would have to socially defer to another person forcontrol over the object, much as would occur if two people in the sameroom were trying to move an object in different directions at the sametime.

[0022] In situations where social conventions for natural manipulationof the virtual object are problematic, controls can be set up to providecontrol of the object by one person or by one location at a time. Thesystem may lockout other locations from then manipulating the object fora period of time thereafter, for example at least one second.Alternatively, the color of all or a portion of the object may change toindicate that a participant has taken “control” of the object as a morevibrant visual cue for other participants to not attempt to move thevirtual object.

[0023] The method also provides for manipulating the object in responseto signals from a voice recognition system integrated with a voiceactivated command structure. The virtual object may change color in itsentirety to indicate that one location has control over the object. Aportion of the object may change color to indicate that one participanthas grabbed that portion of the object as it is being manipulated. Themethod also includes displaying markings, lines or other indicia drawnon the monitor by movement of a participant's finger across the touchsensitive screen or other input device.

[0024] The system can display static computer-generatedthree-dimensional virtual objects. It can also display animatedthree-dimensional virtual objects. In addition to moving andmanipulating the animated object, the users would also be able tocontrol the speed and direction of animation. The three-dimensionalimages and animations can be developed in any typical 3D CAD graphicsapplications and exported into a format suitable for working with thissystem. The format would depend on the type of graphics programminglanguage in higher-level languages used on the computer systems. Suchlanguages are commonly used in sophisticated computer video graphicssystems.

[0025] Further, the objects could be manipulated in other ways such asstretching the objects, changing the color of the objects, actuallydrawing and building the objects displayed on the system. For example,complicated mechanical structures and designs such as airplane wings canbe shown in unfinished format through this video conferencing system.Engineers at remote locations can interact with the virtual object ofthe airplane wing and “on the fly” re-design structural members orrelocate parts. Likewise, scientists at pharmacy companies could usethis invention to model designer drugs and show how drugs interact withenzymes and other bio-molecules.

[0026] While this invention has been shown and described in connectionwith the preferred embodiments, it is apparent that certain changes andmodifications in addition to those mentioned above may be made from thebasic features of this invention. In addition, there are many differenttypes of computer software and hardware that may be utilized inpracticing the invention, and the invention is not limited to theexamples described above. Accordingly, it is the intention of theApplicants to protect all variations and modification within the validscope of the present invention. It is intended that the invention bedefined by the following claims, including all equivalents.

1. A system for manipulating virtual objects displayed on a videoconference broadcast, the system comprising: at least two videomonitors; a video camera associated with each monitor; a manual inputdevice associated with each monitor; and a computer processor systemassociated with each monitor and communicatively connected to a highbandwidth communication network; wherein each processor system displaysa three-dimensional virtual object over the video image on eachassociated monitor, is adapted to receive a signal from each associatedmanual input device and transmit a signal representative of the signalfrom the input device to the other of the processor systems, and tomanipulate the virtual object in response to the signal from the inputdevice and the signal from the other processor system.
 2. The system ofclaim 1 wherein said manual input device is at least one of a mouse, ajoystick, a trackball, a touch screen, or a video gesture recognitionsystem.
 3. The system of claim 1 wherein said manual input device is atouch screen integrated with said video monitor.
 4. The system of claim1 wherein each processor system has a video input card, and each camerais connected to said video input card in each associated processorsystem.
 5. The system of claim 1 wherein the computer processor systemfurther includes voice recognition software for receiving voice commandsfor manipulating the displayed virtual objects.
 6. A system formanipulating virtual objects displayed on a video conference broadcast,the system comprising: at least two video monitors; a video cameraassociated with each monitor; a manual input device associated with eachmonitor; a video processor system coupled to each of said monitor, andcorresponding associated video camera and input device, a computerprocessor system communicatively connected to a high bandwidthcommunication network and each video processor system, wherein theprocessor system displays a three-dimensional virtual object over thevideo image on each monitor and is adapted to receive a signal from eachmanual input device and to manipulate the virtual object in responsethereto.
 7. The system of claim 6 wherein the manual input device is atouch sensitive screen integrated with the video monitor.
 8. The systemof claim 7 wherein the video monitor is a flat panel display ofsufficient size and the associated video camera is of sufficient focallength and positioning with respect to the video monitor to displayapparent life size views of participants from a first location a seenfrom the perspective of participants at a second location.
 9. The systemof claim 6 further comprising a voice recognition system and a voiceactivated command structure for assisting the manipulation of thevirtual object.
 10. A method for manipulating a virtual object displayedon a video conference broadcast at a local and a remote location, themethod comprising: a) generating with a remote processor athree-dimensional image of the virtual object overlaying a first videobroadcast signal from a local video camera for display on a remote videomonitor; b) generating with a local processor a three-dimensional imageof the virtual object overlaying a second video broadcast signal from aremote video camera for display on a local video monitor; c) grabbing aportion of the virtual object displayed at one of the local and remotelocations by placing a real object in close proximity to the portion ofthe displayed image to activate a touch-sensitive screen; and d) movingthe real object while maintaining the real object in active couplingwith the touch sensitive screen; and e) regenerating thethree-dimensional image at each of the local and remote locations tocorrespond to the movement of the real object thereby providing theappearance to viewers at the local and remote locations that the realobject is manipulating a virtual object.
 11. The method of claim 10,wherein said real object is a person's hand, said person located infront of one of said remote or local video monitors and within view ofone of said remote or local video cameras.
 12. The method of claim 11wherein the three dimensional image is repositioned for a movement ofthe real object of at least 0.5 centimeters.
 13. The method of claim 11wherein the three dimensional image is regenerated a sufficient numberof times to provide an animated translation and/or rotation of the threedimensional image.
 14. The method of claim 10 further comprisingdisplaying on the video broadcast signal at least one menu of aplurality of commands in response to a manual or voice input.
 15. Themethod of claim 14 wherein said commands are activated by voice input ormanually touching the command on the display.
 16. The method of claim 10further comprising displaying on both the local video monitor and remotevideo monitors substantially simultaneously the lines corresponding tolines drawn on one of the local or remote video monitors by the tracingof a real object along the touch sensitive screen, wherein the lines areFIGUREs, drawings, letters, numerals or other indicia.
 17. The method ofclaim 10 wherein the portion of the image that is grabbed changes colorsto a predesignated color to indicate that image had been grabbed by anindividual.
 18. The method of claim 10 further comprising the step ofpreventing the image from being grabbed by a second real object whensaid image is moving.
 19. The method according to claim 18 wherein theimage is prevented from being grabbed for a period of at least onesecond after completion of the moving of the image.